Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug in SchedulerJobRunner._process_executor_events #40563

Merged
merged 1 commit into from
Jul 3, 2024

Conversation

vincbeck
Copy link
Contributor

@vincbeck vincbeck commented Jul 2, 2024

#40334 introduced a small bug where the error message is already formatted, hence there is no need to pass parameters to self._task_context_logger.error. That results in this error below:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
  File "/usr/local/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
  File "/usr/local/lib/python3.10/logging/__init__.py", line 678, in format
    record.message = record.getMessage()
  File "/usr/local/lib/python3.10/logging/__init__.py", line 368, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/opt/airflow/airflow/__main__.py", line 58, in main
    args.func(args)
  File "/opt/airflow/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/opt/airflow/airflow/utils/cli.py", line 115, in wrapper
    return f(*args, **kwargs)
  File "/opt/airflow/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)
  File "/opt/airflow/airflow/cli/commands/scheduler_command.py", line 61, in scheduler
    run_command_with_daemon_option(
  File "/opt/airflow/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option
    callback()
  File "/opt/airflow/airflow/cli/commands/scheduler_command.py", line 64, in <lambda>
    callback=lambda: _run_scheduler_job(args),
  File "/opt/airflow/airflow/cli/commands/scheduler_command.py", line 50, in _run_scheduler_job
    run_job(job=job_runner.job, execute_callable=job_runner._execute)
  File "/opt/airflow/airflow/utils/session.py", line 84, in wrapper
    return func(*args, session=session, **kwargs)
  File "/opt/airflow/airflow/jobs/job.py", line 410, in run_job
    return execute_job(job, execute_callable=execute_callable)
  File "/opt/airflow/airflow/jobs/job.py", line 439, in execute_job
    ret = execute_callable()
  File "/opt/airflow/airflow/jobs/scheduler_job_runner.py", line 859, in _execute
    self._run_scheduler_loop()
  File "/opt/airflow/airflow/jobs/scheduler_job_runner.py", line 996, in _run_scheduler_loop
    num_finished_events = self._process_executor_events(session=session)
  File "/opt/airflow/airflow/jobs/scheduler_job_runner.py", line 783, in _process_executor_events
    self._task_context_logger.error(msg, ti, state, ti.state, info, ti=ti)
  File "/opt/airflow/airflow/utils/log/task_context_logger.py", line 170, in error
    self._log(logging.ERROR, msg, *args, ti=ti)
  File "/opt/airflow/airflow/utils/log/task_context_logger.py", line 141, in _log
    task_handler.emit(record)
  File "/opt/airflow/airflow/utils/log/file_task_handler.py", line 229, in emit
    self.handler.emit(record)
Message: "The executor reported that the task instance <TaskInstance: example_hello.hello manual__2024-07-02T19:54:10.890354+00:00 [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally"
Arguments: (<TaskInstance: example_hello.hello manual__2024-07-02T19:54:10.890354+00:00 [queued]>, <TaskInstanceState.FAILED: 'failed'>, 'queued', None)

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Jul 2, 2024
Copy link
Contributor

@RNHTTR RNHTTR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing!

@RNHTTR RNHTTR merged commit a8c4830 into apache:main Jul 3, 2024
52 checks passed
@utkarsharma2 utkarsharma2 added the type:misc/internal Changelog: Misc changes that should appear in change log label Jul 12, 2024
@utkarsharma2 utkarsharma2 added this to the Airflow 2.9.3 milestone Jul 12, 2024
utkarsharma2 pushed a commit that referenced this pull request Jul 12, 2024
utkarsharma2 pushed a commit that referenced this pull request Jul 12, 2024
@vincbeck vincbeck deleted the vincbeck/bug_scheduler branch July 17, 2024 17:30
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:Scheduler including HA (high availability) scheduler type:misc/internal Changelog: Misc changes that should appear in change log
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants